Advanced L2.6KM rootkit development

Pablo Fernandez

Rootkit installation is the phase that divides a compromised computer from an "owned" computer. This article will focus on the development of a rootkit for the 2.6 series of the Linux Kernel. Techniques and methods of hiding the attacker actions within the system will be the primary target, along with discussing how to detect rootkits in the owned box: know your enemy, know thyself.

Knowledge about the internals of rootkits has an enormous value from different points of view; an attacker doesn't really own a system until a proper way to control the entire system has been taken care, a system administrator needs to know how they work in order to be able to evaluate if a system has been compromised.

This article will describe the most important techniques used in a real life extensible LKM rootkit called SIDE, built for the current 2.6 series of the Linux Kernel. In next articles more features will be added to the rootkit.

Hiding the module

Since the rootkit will run within the system as a kernel module, proper care has to be taken so that it's not found with commands such as lsmod or through /proc/modules. Code in listing 1 takes care of this. In order to properly understand how this works, it's important to comprehend how modules are ordered within the kernel and the theory behind this hiding technique (see the Modules frame).

Listing 1. Hiding the module

 
lock_kernel(); /* Held the kernel lock to prevent faulting in SMP systems */ __this_module.list.prev->next = __this_module.list.next;
__this_module.list.next->prev = __this_module.list.prev;
__this_module.list.prev = LIST_POISON1; /* A common practice in kernel development */ __this_module.list.next = LIST_POISON2; /* to invalidate a list that shouldn't be used */ unlock_kernel();

Basically, through this code the module detaches itself from the kernel's internal circular double-linked list of loaded modules.

Hiding processes

The ability to hide processes from every user in the system (including root) is one of the basic futures a rootkit should implement.

User space tools (such as ps(1) or top(1)) know about tasks (processes) reading the /proc directory. Each task running in a system creates an entry in the form /proc/, where useful information about that process can be obtained. What user space tools do is open the /proc directory and query the existence of /proc/, where 1 <= n <= pid_max, if the directory does not exist that PID is assumed to be free, whereas if the directory does exist information can be gathered from it.

With this assumption in mind and the knowledge about the internals of VFS (see VFS internals frame) it's possible to make user space tools believe existent PIDs are actually free. This is done by interrupting the readdir call in the VFS layer. To accomplish this, the table that contains the address of the readdir call has to be modified with the address of the new segment of code that reimplements this function. The porpoise of interrupting readdir is so the filldir argument can be modified to point to a different implementation of filldir, which will discard those directories that identify hidden PIDs.

Listing 2. Fragment of /proc's filldir reimplementation

if (!(process = _atoi(name, &process))); 
/* If this isn't a PID just call the original filldir */ 
else if (!process_is_authed(current) && process_is_hidden(process)) 
/* If process is hidden */      return 0;                                       
/* don't show it (unless current is superroot) */  
if (p_proc_filldir)      
return p_proc_filldir(buf, name, nlen, off, ino, x);

Rootkit detectors

Rootkit detectors use a technique to find hidden processes that consist in sending a SIGCONT signal to all possible processes.

Signals to processes are sent through the kill(2) system call. When a signal is sent to a process that doesn't exist, kill(2) returns the -1 value and errno is set to ESRCH, while when a process does exist kill(2) returns 0.

This way, rootkit detectors realize if processes exists or not without using /proc's data. After this, the list created is compared with the list of processes that /proc shows. If a difference is found between both lists it means that a process is hidden from user space.

From the rootkit point of view, fooling the rootkit detector that relies on this technique is a simple matter of extending the code. All that is required is that the rootkit intercepts the kill(2) system call (see frame System calls) and if a signal is sent from someone else that is not the superroot user (see Superroot frame) to a process which is in hidden state the new callback function should return ESRCH, but if that's the case the original kill(2) function should be called and it's return value returned.

Listing 3. sys_kill() replacement

asmlinkage int new_kill(pid_t pid, int sig) {      struct siginfo info = { .si_signo = sig,
.si_errno = 0, .si_code = SI_USER,
.si_pid = current->tgid, .si_uid = current->uid };
if (!process_is_hidden(pid) || process_is_authed(current))           return kill_proc(pid, sig, &info);
return -ESRCH;
}

Aftermath: The rootkit detector technique

There are several ways for a rootkit detector to recognize rootkits. Since user space tools can be easily fooled, there is no point in keeping rootkit detectors in it. Once in kernel space checking for sys_call_table modifications it's as easy as it can get. Also checking for hidden processes would be easier, since they can't be unattached from the processes list as easy as modules; their presence in such list is the only warranty that they will get execution time slices.

While some rootkit detectors do use some of this techniques, most of them stay in user space. The lesson for paranoid administrators is not to blindly rely on rootkit detectors.

Hiding network connections

User space programs and applications know about ongoing network traffic through the /proc/net entries, within that location several entries (that look like files, but are actually proc_dir_entry structures), like tcp and tcp6 (if CONFIG_IPV6 is enabled). This entries contain the information of networking that's going on in the system.

Using a different method read calls can be intercepted too, thus the information returned can be modified in transit. While the network connection is still active (or a socket is in LISTEN state) netstat and tools alike won't be able to see it.

SIDE provides a very nice way of hiding network connections, very similar (although not remotely as powerful) as Netfilter. A list of conditions and commands is defined during runtime (see Table 1) and when information about sockets is requested the list is matched with each socket, if some rule applies the command associated with the condition is executed. The command can be either to show or hide the socket from user space. This list is pretty powerful. Default actions can be defined with the all condition at the end of the list, which can be fully manipulated during runtime (and it can even execute default commands when the module is loaded).

The method used to hide network connections consists in grabing a handle to the proc_dir_entry of the protocol that needs to be intercepted, such as tcp. proc_dir_entryies belonging to the /proc/net space can be found through the circular double linked list in proc_net->subdir. Iterating through it checking for the correct name in the node->name member should get it right.

Once this structure has been grabbed, the seq_show pointer needs to be overwritten with a new implementation of this function. SIDE's implementation fetches the correct data (using the original function) and matches each line of the data with the loaded rules, applying specified actions in the matching lines.

The problematic

Due to the nature of network traffic it's impossible to really hide activity from the eyes of a wise administrator. There are usually several hops between the compromised system and the other end of the hidden communication. These hops will clearly show this hidden traffic and there's little to do about it.

In fact, even when a connection is not yet established, if a socket in LISTEN state is hidden, nmap will reveal that there's an open port which netstat doesn't show. Certainly something that can be done to avoid this problem is using Port Knocking (hakin9 6/2005), but once the connection is opened and traffic is on the way it will be easy to detect this traffic (Removing spiderwebs - detection of illegal connection sharing, hakin9 3/2005).

A neat trick would be not to establish a connection at all, exchanging packets of information in ICMP or UDP packets is an interesting way of controlling a system and to get information about it's state. This way the attacker can even control the compromised system without leaving any trails, using spoofed UDP or ICMP packets.

The present rootkit will be expanded in subsequent articles that will enhance it with many more features, including those mentioned here.

Hiding files

A system is often compromised to be used as a safety platform from where to launch [D]DoS attacks, or to be used as a hop between the attacker and another compromised system. Most of the time the attacker will most likely need to upload some files to the system to perform these attacks. Of course, if such tools are found by the administrator, questions will arise. Thus, a rootkit should always be able to hide files in the system, again, from any user, including root.

Again, we are going to use the knowledge of VFS to perform such actions, using the exact same method used to hide processes.

This time the fs object will store a list of hidden filenames. This filenames lack a path, so any file hidden in a directory will imply hiding every file with the same filename in all directories. The reason for this feature is to force the superroot user to use non standard names since there are still other methods that are not being handled, for example, even though hidden files are not going to be listed in directories they are still accessible through system calls such as open(2), stat(2), etc. SIDE does not currently support hiding files from this methods, although all it's required is to replace those system calls (and a few more such as rename(2)). Suggestion: it would be a nice exercise to expand this features when you finish reading this article.

Note that the methodology used is filesystem dependant. If hiding files in different mount points is desired SIDE has to be modified in the vfs.c file to include hiding from those mount points. It's completely safe to use the same readdir and filldir as the root filesystem uses.

The problematic

Again, the action of hiding files has an intrinsic problematic on it's own, similar to the problem with network connections.

Files are stored in disks, disks that can be accessed in different ways, like a rescue CD-ROM where the user mounts the root partition. Since the rootkit isn't loaded in the running kernel, the hidden files are not hidden any more. There are different ways to make it a little harder to find those hidden files. The easiest and most frequent protection is security by obscurity; storing files in non-standard places with non-descriptive and confusing names.

Another and much better approach is storing all the files in a loopback filesystem, of course, given this methodology proper care should be taken so that such mounted filesystem is seen neither with mount(8) nor through /proc/mounts. This methodology also allows easily encryption of the filesystem, so that even if the administrators are suspicious, they will never get to see what's inside of the filesystem.

Normal user with root permissions

The superroot user is identified by a special non-zero UID and GID, thus the superroot user lacks root permissions, of course, this is absolutely unacceptable. That's why SIDE implements a mechanism to establish an UID of 0 to every process the superroot executes.

This is also done in the interrupted call to the lookup() function in the /proc directory. Whenever an authenticated process (see the Superroot frame) access anything in it, it's UID and other related values are set to 0 (root), and some capabilities are fully activated (cap_effective, cap_inheritable and cap_permitted). If the process does not access anything in /proc it will run with the Superroot UID, that's why some extremely little and simple programs will not identify themselves as root, such as the whoami command.

Runtime usability

SIDE provides a very comfortable interface during runtime. In vfs.c, the rootkit intercepts lookup() calls in the /proc filesystem. This way, the superroot user (see Superroot frame) can interact with the rootkit, to make it, for example, hide a process or to grant root (or superroot) permissions to some user. Sending commands to the rootkit it's pretty easy, all the user has to do is try to access a file in the /proc filesystem. The name of the file will be interpreted as the command.

SIDE organizes commands by objects, thus, depending on what the user wants to do, commands are executed against specific objects.

Currently SIDE recognizes three different objects: net, for manipulation of the network list; sys, to handle process and user related properties; fs, for manipulation of the hidden files list.

There are several commands for this objects. For brevity sake a few of them will be listed in Table 1. The rest of them can be found in the COMMANDS.txt file within the package.

Commands should be executed like the following:

echo > /proc/[object].[command[=args]]

Modules

Loaded modules are stored within the kernel in a circular double-linked list, where each node of the list represents a struct module (defined in include/module.h). Modules are usually gathered reading the /proc/modules entry. The list is of modules is created by m_show, in kernel/module.c, going through the previously named linked list.

The fact that this list is accessible from each module allows it modification and manipulation. The tecnhnique used to hide the module is to detach the module node from the linked list, connecting directly the previous and next value within the list with themselves.

This technique has been covered in-depth by a very interesting article written by Mariusz Burdach in hakin9 3/2005.

System calls

System calls are the interface that reside in the kernel, which communicate user space with the kernel. Everything that goes from the kernel to the user space or vice versa has to go through a system call. This is the main reason rootkits had always been specially interested in intercepting them, since controlling them translates into controlling what the user sees and what the user can do.

There are many system calls, like open(2), read(2), etc. All this functions are referenced by pointers in an array called System Calls Table, better known as sys_call_table.

Historically modifying the sys_call_table was a matter of just overwriting the desired pointer with a new memory address with the new function, this way any system call (except for execve(2) which needs to be handled more carefully) could be easily intercepted.

Figure 1. Modification of sys_call_table

(Un)fortunately, since 2.5.41 Linux does not export the sys_call_table symbol any more, while it still exists the memory address is no longer available to modules.

To find the correct address there's a technique that can be used: /usr/src/linux/include/asm/unistd.h lists the order of the sys_call_table with __NR constants that define the offset in the array where each system call is, and since the address of each system call is known (every system call is exported), memory can be scanned looking for the sys_call_table.

The code that does this simply declares a pointer that starts at a low memory address (usually the address of loops_per_jiffy is used) and loops until an __NR offset of the pointer matches the correct address of the same system call. If a high memory address is reached (like boot_cpu_data), something wrong happened (maybe a module is already intercepting the system call which is being used to look the sys_call_table), which means the system call table could not be found. If the system call table is not found, interruptions of system calls wouldn't be possible. Note that this is completely independent of the VFS interruptions.

Listing 4. Search for sys_call_table

unsigned long ptr;
extern int loops_per_jiffy;
for (ptr = (unsigned long) &loops_per_jiffy; ptr < (unsigned long)&boot_cpu_data; ptr += sizeof(void *)) {      unsigned long *p;
p = (unsigned long *)ptr;
if (p[__NR_close] == (u32) sys_close) /* When this condition is met p points to sys_call_table */           return (u32 **) p;
}

Once sys_call_table has been found altering it is as simple as it was in the past, see Listing 3 for an example of how to replace the open(2) system call.

Listing 5. Intercepting a system call

u32 **sys_call_table;
asmlinkage int (*old_open)(const char *, int, mode_t);
if ((sys_call_table = find_sys_call_table())) {      old_open = (void*) sys_call_table[__NR_OPEN];
sys_call_table[__NR_OPEN] = (u32*) new_open;
}

It's important to keep in mind that system calls are very critical to a system. If a system call is intercepted and the new callback function (in Listing 3, new_open) doesn't behaves correctly the system will misbehave too, and most probably become completely unstable. A common practice is to call the original system call from the new callback function when the new callback function decides to allow it's execution. That's exactly the reason code in Listing 3 saves a pointer to the original function, also, when the module is to be unloaded the sys_call_table should be modified to point to the original location.

VFS Internals

The Virtual File System or Virtual Filesystem Switch is a layer between file-related system calls (like open(2)) and the actual filesystem implementations (like ext2, ext3, reiserfs, jfs, etc.). It provides a common interface to ease the work of filesystem implementators.

Filesystem implementations have to define a set of predefined functions and methods, and notify the VFS layer about those methods, which are invoked by it as callback functions through function pointers. Basically VFS has an structure that each filesystem has to fill and register in the VFS layer. In that structure it is given the required information to find the addresses where those callback functions are to be found.

Throughout the development of a rootkit, the most interesting call will undoubtedly be readdir. This callback function provides the algorithm to call the function parameter filldir, which will be called for each file or directory read by readdir. Its return value is used to prepare the information about the directory read.

Throughout this article a technique is going to be heavily used which uses the return value 0 in the filldir function. This return value causes readdir to discard the information of the readed item.

Figure 2. The VFS layer

Superroot

If any user of the system was able to control a rootkit that has the ability to bring the system down to it knees, it wouldn't be a very good one. Before SIDE executes any command users must authenticate themselves to the rootkit, this is done with the sys.superroot command. For this command to successfully authenticate the user, the key has to be specified as a parameter of the command.

The key is a (usually) random string that identifies the installation. SIDE selects the key for each installation when the configure script is run.

When the correct key is given the UID and GID of the user is changed to those that identify the superroot user (selected at configure time as well).

The superroot user is required to allow the execution of commands and to avoid hiding information from that particular user that is hidden to other users.

Table 1. Command list

Conclusion

Throughout this article different techniques and approaches were studied on the subject of rootkit development in the 2.6 series of the Linux Kernel. Methodologies on how to hide network connections, processes, modules and files were reviewed and counter measures that rootkit detectors use, as well as new counter measures that should be put in practice by rootkit detectors developers and administrators. Yes.